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(57) Digital waterniarking of data, including image, 
video and audio data, is performed by repeatedly insert- 
ing the watermark Into subregions or subimages of the 
data. Similarly, the watermark is repeatedly extracted 
from the subregions of the data. 
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E>escf1ptlon 

The present invention relates to cfigital watenmark* 
ing of data including image, video and multimecfia data. 
Specifically, the invention relates to the Insertion and 
extraction of eni>edded signals for purposes of water- 
niarking, in which the insertion and extraction proce- 
dures are repeatedly applied to subregions of the data. 
When these subregions correspond to the 8x8 pixel 
blocks used for MPEG and JPEG compression and 
decompression, the watermarking procedure can be 
tightly coupled with these compression algorithms to 
achieve very significant savings in computation. 

The proliferation of dgitized media such as Image, 
video and multimedia is aeating a need for a security 
sy^em which facilitates the identification of the source 
of the material. 

Content providers, i.e. owners of worte in cfigital 
data form, have a need to embed signals into 
video/image/muttimedia data which can sul^sequentty 
be detected by software and or hardware devices for 
purposes of authenticating copyright ownership, control 
and management 

For exanrple, a coded signal might be inserted in 
data to indicate ttiat the data should not be copied. The 
embedded signal shouM preserve the image fidelity, be 
robust to common signal transformations and resistant 
to tampering. In addition, consideration must be given to 
the data rate that can be provided by the system, 
though current requirements are relatively low - a few 
bits per frame. 

In EP-A-766 468 corresponding to U.S. Patent 
Application Serial No. 08/534,894. filed September 28. 
1995. entitled "Secure Spread Spectrum Watermarking 
for Multimedia Data" and assi^ed to the same 
assignee as tie present invention, which is incorporated 
herein by reference, there was proposed a spread spec- 
trum watermarking method which embedded a water- 
mark signal into perceptually significant regions of an 
image for the purposes of identifying the content owner 
and/or possessor. A strength of this approach is that the 
watermark is very difficult to remove. In fact, this method 
only allows the watermark to be read rf the original 
image or data is available for comparison. This ^ 
because the original spectrum of the watermark is 
shaped to that of the image tiirough a non-linear multi- 
plicative procedure and this spectral shaping must t>e 
removed prior to detection by matched filtering and the 
watermark is inserted into the N largest spectral coeffi- 
cients, the ranking of which is not preserved after water- 
marking. Thus, this method does not allow software and 
hardware devices to cfirectiy read embedded signals. 

In an article by Cox et a!., entitied " Secured Spec- 
trum Watermarking for Multimedia" available at 
httpyAvww.neci.nj.com/tr/index.html (Technical Report 
No. 95-10) spread spectrum watermarking is described 
which embeds a pseudo-random noise sequence into 
the digital data for watermaridng purposes. 



The above prior art watermark extraction methodol- 
ogy requires tiie original image spectrum be subtracted 
from the watermaric image spectrunt This restricts tiie 
use of the metfKxJ when there is no origir^ image or 
5 original image spectrum available. One application 
where ttiis presents a significant diffk:ulty \s for third 
party device providers desiring to read embedded infor- 
mation for operation or denying operation of such a 
device. 

10 In U.S Patent No. 5.319.735 by R.D. Preuss et al 
entitied 'Embedded Signalling" digital information is 
encoded to produce a sequence of code symtx>ls. The 
sequence of code symbols is embedded in an audio sig- 
nal by generating a corresponding sequence of spread 

IS spectrum code signals representing the sequence of 
code symbols. The frequency components of the code 
signal being essentially confined to a preselected sign- 
aling barxJ lying within the bandwidth of the audio signal 
and successive segments of the code signal corre- 

20 spends to successive code symbols in ttie sequence. 
The aucfio signal is continuously frequency analyzed 
over a frequency band encompassing ttie signalling 
band and the code signal ^ dynamically fOtered as a 
function of the analysis to provide a modified code sig- 

25 nal with frequency component levels which are, at each 
time instant, essentially a preselected proportion of ttie 
levels of the audio signal frequency components in cor- 
responding frequency ranges. The modified code signal 
and the audio signal are combined to provide a compos- 

30 ite audio signal in which the digital information is 
embedded. This conponent audio signal is tiien 
recorded on a recording medium or is otherwise sut>- 
jected to a transmission channel. Two key elements of 
this process are the spectral shaping and spectral 

35 equalization tiiat occur at the insertbn and extraction 
stages, respectively, thereby allowing the embedded 
signal to be extracted without access to the unwater- 
marked original data. 

In EP-patent application No. 97 11 53 53.1 cone- 

40 spending to U.S. Patent Application Serial No. 
08/708,331 . filed September 4, 1 996, entitied "A Spread 
Spectrum W^ermaric for Embedded Signaling" by Cox, 
and incorporated herein by reference, there is 
descrit)ed a mettiod for exbacting a watermark of 

45 embedded data from watermarked Images or video 
wittiout using an original or unwatermarked version of 
the data. This work can be viewed as an extension of 
the original work of Preuss et al from the audio domain 
to images arxJ video. 

50 This method of watermarking an image or image 
data for embedding signaling requires that the DCT 
(discrete cosine transform) and its inverse of the entire 
image be computed. There are fast algorithms for com- 
puting the DCT in A/ log /V time, where A/ is the number 

55 of pixels in the image, htowever, for A/=512x512. the 
computational requirement is still high, particulariy if the 
erxxxiing and extracting processes must occur at video 
rates, i.a 30 frames per second. This method requires 
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approximately 30 times the computation needed for 
MPEG-II decompression. 

One possS>le way to achieve real-time video water- 
marking is to only watermark every frama However, 
content owners wish to protect each and every video s 
frame. Moreover, if it is known which frames contain 
embedded signals, it is simple to remove those frames 
vnth no noticeable degradation in the video signal. 

In U.S. Patent Application Serial Na 08/715.953, 
filed September 19. 1996. entitled "Watermarking of 
Image Data Using MPEG/SPEG Coeffidents" by Cox. 
and incorporated herein by reference, there Is 
desaO:>ed an alternative method, which is to insert the 
watermark into n x n blocks of the image (subimages) 
where n«N Then the corrputation cost is 

N 

-n log n = N\oQn, 



For N=512x512=2^^ and n=dx8=2^. the asymp- 
totic saving is only a factor of 3. However, empirically the 
cost of computing the DCT over the entire image may 
be significantly higher when cache, loop unfokling and 
other efficiency issues are considered. Thus, the practi- 
cal difference may approach a 30 fokl savings. More 
importantly, if the tHock size is chosen to be 8x8, i.e. the 
same size as that used for MPEG image compression, 
then it is possible to tightly couple the watermark inser- 
tion and extraction procedures to those of the MPEG 
coirpression and decorrpression algorithnris. Consider- 
able computatk>nal saving can then be achieved since 
the most expenses conputations relate to the calcula- 
tion of the DCT and its inverse and these steps are 
already conputed as part of the compression and 
decompression algorithm. The inaemental cost of 
watermarking is then very small, typically less than 5% 
of the corrputational requirements associated witti 
MPEG. 

The present invention improves the reliability of the 
invention described in the 08/715.953 application by 
storing watermark information into subimages. and 
extracting watermark information from siA)images. in a 
manner different from tiiat described earlier. 

The present invention improves the reliability of the 
prior systems by systematically varying tiie order in 
which watermark signal corrponents are inserted into 
each subimage, by inserting only part of the watermark 
signal into each subimage, and, during watermark 
detectioa by combining the watermark signals found in 
groups of sutMmages to reconstruct the original water- 
mark signal before testing for correlation with any pre- 
defined watermarks 

For detection, a reverse transformation is applied to 
each sibimage to reconstruct ttie watermark informa- 
tion that was stored in that subimaga The resulting sig- 
nals are then averaged together to reconstruct the 
whole watermark, and to reduce noisa Rnally, this 



reconstructed watermark is corrpared against a prede- 
fined set of watermark signals to determine which one 
was inserted into the image. 

A principal object of the present invention is there- 
fore, the provision of inserting a sut>set of a watermark 
into a sii>set of subregions of data to be watermarked. 

Another object of the invention is the provision of a 
digital watermarking system in which a vtratermark \s 
extracted by averaging the watermarked signal from 
subregions of watermarked data, and then correlating 
the resulting signal to determine the watermark. 

A further object of the invention is the provision of a 
digital watermarking system in which the watermark is 
composed of two portions, a verification portion and a 
synchronization portion, in order to irrprove watermark 
extraction reliability 

Further and still other objects of the invention will 
become more dearly apparent when the following 
description is read in conjunction with the accompany- 
ing drawing. 

Brief Description of the Drawings 

Rgure 1 is a schematic block diagram of watermark 
insertion procedure; 

Rgure 2 is a schematic bkx^k diagram of a water- 
mark insertion procedure in accordance witti the 
teachings of the present invention; 
Rgure 3 is a schematic bkx:k diagram of a water- 
mark extraction procedure; 
Rgure 4 is a schematic block diagram of a water- 
mark extraction procedure in accordance witii the 
teachings of the present invention; 
Rgure 5 is a graphic representation of a zigzag pat- 
tem useful for vectorizing subimages; 
Rgure 6 is a graphk: representation of rotation of 
PN sequences; 

Rgure 7 is a graphical representation of an 8x8 
block shewn the spatial relation of averaged terms; 
Rgure 8 is a schematic block diagram of a method 
for inserting watermarks in accordance with the 
present invention; arxi 

Rgure 9 is a schematic block diagram of a method 
for extracting watermarks in accordance with the 
present invention. 

Detailed Description 

Referring now to the figures, arxi Rgures 1 through 
4 in particular, there is shown schematic block diagrams 
of a general mettiod fa inserting and detecting water- 
marks in digital data, for instance images. 

In the follcwing description, reference may be made 
to image data or images. While the invention has appli- 
cability to image data and images, it will be understood 
tfiat the teachings herein and the invention itself are 
equally applical^le to video, image and multimecfia data 
arxJ the term Image" and *1mage data" will be urxier- 
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stood to include these terms wtiere appGcable. As used 
herein, "^ermark" wfll be understood to indude 
enrt>edded data, symbols, images, instructions or any 
other identifying information. 

In the following description, reference is made to 
procedures descrQ>ed in U.S. Patent Application Serial 
No. 08/534.894 for inserting and extracting or delecting 
a watermark in images as INSERT-ORIGINAL and 
EXTRACT-ORIGINAL, respectively. Reference is made 
to procedures described in U.S. Patent Application 
Serial No. 08/708.331 filed September 4. 1996 for 
inserting and extracting a detecting watermarks in 
images as INSERT-WHOLE and EXTRACT-WHOLE, 
respectively. And reference is made to procedures 
desaibed in U.S. Patent Application Serial Na 
08/715.953 for inserting and extracting or detecting 
watermarks in images as INSERT-MPEG-A and 
EXTRACT-MPEG-A. respectively. 

Rgure 1 shows a schematic block diagram of 
INSERT-WHOLE procedure for inserting watermarks 
into inriages. The watermark signal, in the form of a finite 
sequence of syntete chosen from an alphabet, is pro- 
vided as an input to an error conection encoder 10 
which transfbrn^ tilts sequence into another sequence 
that contains redundant information. The output of 
encoder 10 is provided to a PN-mapper 1 1 , which maps 
each symbol of the encoded watermark into a pre-spec- 
rf led pseudo-random noise (PN) code. The output of the 
PN-mapper 1 1 is provided to a spectral transfbnmer 12. 
which converts the pseudo-random noise sequence into 
the frequency domain. The conversion preferably is by 
discrete cosine transform (DCT), however, fast fourier 
transform, wavelet type decomposition and the like may 
also be used for frequency conversion. Concurrentiy. 
the data to be watermarked is provided to another spec- 
tral transformer 13. The outputs of the two spectral 
transformers 12 and 13 are then provided as inputs to a 
spectral shaper 14. which modifies the spectral proper- 
ties of the pseudo-random noise codes from spectral 
transformer 12 to mask the watermark when added to 
the image data The spectrally transformed data to be 
watermarked, from spectral transformer 13. is also pro- 
vided as an input to a delay 1 5. The output of the spec- 
tral shaper 14 is tiien added to the output of delay 15 at 
a summer 16. The summer output is subject to an 
inverse transform 17. TTie result of the inverse transform 
is watermari<ed data. 

INSERT-MPEG-A differs from INSERT-WHOLE by 
segmenting the data to be watermarked into multiple 
blocks, such as 8x8 pixel subimages or sutnegions. 
Each block of data then has the watermark inserted 
according to the atxxve descrbed metfxxJ. That is, for 
each 8x8 subimage or subregion, a pseudo-random 
numbo' (PN) sequence is inserted into the DCT coeffi- 
cients after suitable spectral shaping. The procedure is 
repeated for all such subin^ges or subregions. The size 
of the subimage or sutxegion is preferattly 8x8. but it 
can be of other sizes, such as 2x2. 3x3, 4x4 or 16x16. 



Rgure 2 shows a schematic block dia^am of a 
watermark insertion procedure in accordance with 
teachings of the present invention. The watermark sig- 
nal is processed into a noise spectrum signal by the 

5 error correction encoder 20. the PN mapper 21 , and the 
spectral tran^rmer 22. in the same manner as 
described in conjunction with Rgure 1. However, unlike 
INSERT-WHOLE or INSERT-MPEG-A, the watermark \s 
then used as an input to a watermark segmenter 23, 

70 which systematically separates the watermark into sev- 
eral sut>waternnarks. Any portion of the original water- 
mark might appear redundantly in several of the 
resulting subwatermarks. Concurrently, the data to be 
watermarked is used as an input to data segmenter 24. 

15 which segments the data into blocks or subregions. 
such as 8x8 subimages. as in INSERT-MPEG-A. Each 
of the subwatermarks output by the watermark seg- 
menter 23 is then inserted into a data block by one of 
the watermark inserters 25a. 25k>. etc. TTie procedure 

20 used by the watermark inserters 25a, 25b. etc., is the 
same procedure desaik>ed connection with watermark 
inserter 18 in Rgure 1. That is. each subwatermark is 
added into a spectrally transformed data block after 
spectral shaping, and the resulting data is then trans- 

2S fbnned t>ack into the spatial domain. Rnally, the water- 
marked data blocks from the watermark inserters 25a, 
25b. etc., are assembled by data connbiner 26 to pro- 
duce watermarked data 

Rgure 3 shows a schematic bkx:k diagram of the 

30 EXTRACT-WHOLE procedure. The watennarked 
image, video or multimedia data is first used as input 
into a spectral normalizer 30 to undo any previously per- 
formed spectral shaping. If the data contains a water- 
mark, then the output of the spectral normalizer 30 will 

35 reserrt^le the spectral trar^formation of the PN coding 
of that watermark (the signal that was input to the spec- 
tral shaper 14 in Rgure 1). The output of the spectral 
normalizer 30 is then used as an input to several corre- 
lators 31a. 31b, etc.. which test the watermark with the 

40 PN codes used to represent tiie various symbols that 
the encoded watermark might contain (i.a each con^ela- 
tor tests tor one PN code tiiat is used to encode a sym- 
bol by the PN mapper 1 1 of Rgure 1). The outputs of the 
correlators 31a, 31b, etc., are used as inputs to a deci- 

45 sion circuit 32, which determines the most likely 
sequence of symbols. Rnally, this sequence is cor- 
rected by an. error corrector 33. which performs the 
inverse coding that was performed by the error confec- 
tion encoder 10 in Rgure 1. The result is the extracted 

so watermark. 

In EXTRACT-MPEG-A. the data from which a 
watermark ^ to t>e extracted is first segmented into sev- 
eral blocks, such as 8x8 subimages. exactly as in 
INSERT-MPEG-A. The signal from each subimage is 

55 then normalized and used as input into a bank of con'e- 
tators similar to the correlators 31a, 31b. etc. in Rgure 
3. The output from the con^elators is then averaged with 
the outputs of corresponding correlators from other 
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subimages. and the resulting average con-elations are 
used as inputs into the decision circuit 32 for sut>5e- 
quent processing as descrS>ed atxiva 

Figure 4 shows a scheniatic block cfiagram of a 
watermark extraction procedure in accordance with the 
teachings of the present invention. The watermarked 
data is f irst segmented into tilocks by data segmenter 
40, which conesponds to the data segmenter 24 used 
during the insertion procedure in Rgure 2. Each of the 
data blocks is provided to a respective spectrum nor- 
malizer 41a. 41b. etc. to produce a signal resembling 
the subwatenmark that was inserted into the respective 
data block. These inserted subwatermark signals are 
then i^ed as inputs into a watermark comt>iner 42. In 
the combiner 42. parts of the watermark that appear 
redundantly in several subwatermarks are averaged 
together to reduce noisa The output of the watermark 
combiner 42 is provided as the input to a symbol sepa- 
rator 43 which divides the watermark into parts, each of 
which corresponds to one synix)l from the encoded 
watermark signal (the output of error correction encoder 
20 in Rgure 2). 

These syrTtx>ls from separator 43 are provided as 
inputs to respective watermark identifiers 44a. 44b etc. 
each of which includes of a bank of correlators and a 
decision circuit, as shown in Rgure 3. The outputs of the 
watennark identifiers are symbols from the atphak>et 
used in the original encoded watermark signal. The 
identified syntx>ls are reassembled into a complete 
encoded watermark by the symtx>l combiner 45. Rnally. 
the resulting encoded watermark is decoded by the 
error corrector 46. 

The insertion and extraction procedures will now be 
descrbed in more detail. In INSERT-ORIGINAL and 
EXTRACT-ORIGINAL, the object is to embed a single 
PN (pseudo random number) sequence into an image 
when the original image is available at the time of 
extraction. The irtformation associated with the PN 
sequence is assumed to be stored in a database 
together with the original image and the spectral loca- 
tion of the embedded watermark. The locations of the 
watermarked corrponents has to be recorded because 
the implementation approximated the N perceptually 
most significant regions of the watermark by the N larg- 
est coefficients. Hcwever, this ranking was not invariant 
to the watermarking process. The N largest coefficients 
may t>e Afferent after inserting the watermark than 
before inserting the watermark. 

In order to avoid this problem, the present invention 
places a watemwk in predetermined locations of the 
spectrum, typically the first N coefficients. However, any 
predetermined locations couki be used, though such 
locations should bebng to the perceptually significant 
regions of the spectrum if the watermark is to survive 
common signals transformations such as compression, 
scaling, etc. 

More generally, the information to be embedded is 
a sequence of m symtx}ls drawn from an alphabet A 



(e.g. the binary digits or the ASCII symbols). This data 
is then supplemented with additional symbols for enor 
detection and conection. Each syntol is then spread 
sf^sctrum modulated, a process that maps each symbol 

5 irtto a unique PN sequence known as a chipi The 
number of bits per chip is preset - the longer the chip 
length, the higher the detected signal-to-rK3ise ratio will 
be. but this is at the expense of signaling bandwidth. 
The power spectrum of the PN sequence is white, 

10 i.e. flat, and is therefore shaped to match that of the 
"noise", i.e. ttie imageMdeo/audio^or multimedia data 
into which the watermark is to be embedded. It is this 
spectral shaping that must be modified from the prior 
methods so that the extraction process no tonger 

15 requires the original image. To do this, each coeffk:ient 
of the watermarked spectrum is scaled by the local 
average of the power in the image spectral coeffk;ient 
rather than the coefficient itself, i.e. 

20 f]=fi'^aav9[\ffQWf (1) 

The averaging is the averaging of the absolute coef- 
ficient values and not the coeff icient values themselves. 
Th^ is effectively estimating the average power present 

25 at each frequency Other averaging procedures are pos- 
sible, for example, averaging over several frames or 
average of local neightorhoods of 8x8 blocks. 

This average may be otstained in several ways. It 
may be a local average over a two dimensional region. 

30 Alternatively, the two dimensional spectrum may be 
sampled to form a one dimensional vector and a one 
dimensional local average may be performed. One 
dimensional vectorization of the two dimensional 8x8 
DCT coeffidents is already performed as part of MPEG 

35 il.The average may be a simple box or weighted aver- 
age over the neighborhood. 

For video data, temporal averaging of the spectral 
coefficients over several frames can also be applied. 
However, since several frames are needed for averag- 

40 ing at the spectral normalization stage of tiie extractor, 
the protection of individual video frames taken in isola- 
tion may not be possible. For this reason, the present 
invention treats video as a very large collection of still 
images. In this way, even individual video frames are 

45 copy protected. 

In order to extract the watermark, it is necessary to 
perfbnm the spectral normalization, in which the previ- 
ously perfomied spectral shaping procedure is inverted. 
In the present invention, the original unwatermarked 

50 signal is not available. Thus, the average power of tiie 
frequency coefficients. aug(|^D,is approximated by the 
average of the watermarked signal, i.e. avg{\fi\) 

avg(|^D«av^(K>D (2) 

55 

This is approximately true since aavgQff])Wf«ff, 
where Wj is the watermark component, and a is a con- 
stant typically in the range between 0.1 and 0.01. 
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The normalization stage then divides each coeffi- 
cient (// ) in the received signal by the local average avg 
(|f/D in the neighborhood. 

That is. 



avg(IM) " av9(|M) av3(|/,D 

The first term, on the right haixl side (RHS) of 
Equation (3) , 



is considered a noise term. This term was not present in 
the system desaa>ed U.S. Patent Application Serial Na 
08/534,894. because access to the unwatermarked 
coefficients allowed this term to be removed. The sec- 
ond term a Wi is the original watermark signal which can 
new be detected using conventional correlation. 

If the watermark Is extracted from any single 8x8 
block, the detector reliability is very low. If, however, the 
watermarks extracted from each 8x8 block are first 
added together arxl the averaged watermark is then 
applied to the con-elator. then a very strong and unanv 
biguous response is obtained. This differs from the 
method desaibed in U.S. Patent Application Serial Na 
08/715,953 in which conelation occurred within each 
t>lock and the output from each correlator was averaged 
together. The present invention was found to improve 
the detection response and significantly reduced the 
computation requirement associated with each block. 

In practicing the present invention preferably there 
is a unique PN sequence for each symbol in the alpha- 
bet The mettxxj is relatively robust to clipping since the 
detector output reduces linearly with the quantity of 8x8 
sut^image blocks in the imaga For DVD (digital video 
disk) embedded signaling for APS (anak)g protection 
system) and CGMS (copy generation management sys- 
tem), there would be a total of 8 or 16 PN sequences. 

The number of 8x8 blocks in a 512x512 image is 
4096, suggesting that significantly more than one of 16 
symbols can t>e embedded in an image or video frame. 
Assume, for example, that it is desired to embed 1 out of 
128 symbols in an image. It is necessary to perform 128 
parallel correlations. This is corrputationally tractable 
but hardware implementations of each correlation 
t>ecome rrxjre complex. An atternative method is to only 
use two binary symbols. It may be preferable to associ- 
ate more than one PN sequence with each of the two 
binary syntx)ls or bits in order to increase the difficulty 
of intentionally renxiving the watermark. In this case, 
there are only two correlators and a binary string may 
be erTt>edded into the image. The raw bit error rate will 
be very high due by the low detector output However, 



this can be reduced to acceptable levels by using error 
correcting codes, such as Reed-Solomon (RS). RS 
codes are robust to burst error which may occur 
because of clipping of tfie imaga Other error correcting 

5 codes may also be used. 

When i^ing this metfKx). it is necessary for the 
receiver to know the start location of the encoded block. 
The start location may not be obvious, particularly when 
the image has been subjected to clipping. However. 

TO convention synchrortizing methods can be used; such 
as preceding each block with a special or unique sym- 
bol or string of symbols. 

To insert a watermark, each 8x8 block is treated as 
an individual sutM'mage or subregion. Tile DOT of the 

IS subimage e then computed and the two dimensional 
DOT is vectorized in the zigzag pattern shown in Rgure 
5, although other patterns are also possible. These two 
stages constitute most of ttie calculations but are part of 
the MPEG encoding process. Next, a PN noise 

20 sequence {w, ...W,^ is irrserted into tiie DOT coeffictenis 
using Equation 1 as befora The length of the PN 
sequence cannot exceed 64 (in an 8 x 8 block) and is 
typically much shorter, in the range of 1 1 to 25. If only a 
single code is to be inserted into the image, then the 

2s same PN sequence is inserted into each of the 720 x 
480/64 - 5400 blocks. However, a variation may be per- 
formed at this point in the procedure. Within each row of 
blocks, ttie PN sequence is cyclically rotated by one fre- 
quency coefficient prior to insertion in the subsequent 

30 block. Similarly, the PN sequence is cyclically rotated by 
one frequency coefficient at the start of each new row. 
Rgure 6 illustrates an order of rotations. 

The purpose of these rotations or shifts is to 
improve the response of the watermark extraction 

35 stage. Earlier experiments revealed that certain DOT 
coefficients were more difficult to estimate than others. 
The location at these coefficients varied from image to 
imaga However, within an image, the coefficient could 
be consistentiy poor. Consequentiy. without shifting. 

40 one or nrxxe of the estimated watermark coefficients 
oouM be signifk;antly degraded r^tive to the other 
watermark coefficients, thereby reducing tiie detector 
perfomnance. Conversely, shifting significantiy reduces 
the effect a poor DOT coefficients has on a single water- 

45 mark coefficient arxJ the detector performance is mark- 
edly improved. Note that any cyclic pattern can be used. 

Further modifications are useful once rotation of the 
watermark has been introduced. Rrst. the length of the 
watermark may now be significantiy greater than 64. 

50 Then, for each block only a small sul)set of tiie water- 
mark (say five) coefficients is inserted into the first five 
OCT coefficients (exducGng the d.c. term). Because of 
tiie rotation, a different sut)set of tiie watermark is 
inserted into neightx)ring 8x8 blocks. Rnalty. having 

55 completed the watermark insertion, the MPEG encoder 
is at)le to proceed with the sut>sequent stages of com- 
pression. 

Note that the watermark may also t>e inserted after 
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the MPEG quantization stage to reduce distortion of ttie 
watermark. MPEG-2 perfornis a convenient one dimen- 
sion vectorization caOed "zigzagging", which allows a 
simple 3x1 txix average to be performed on the coeffi- 
cients (exducGng the d.a term). 

In practice, performance ms Improved if the aver- 
aging is performed using the 2 four -connected coeffi- 
denls closest to the d.c. term, as illustrated in Rgure 7. 
i.e. the two coefficients above and to the left 

Watermark detection begins by first extracting the 
FN noise sequ&ice from each 8x8 t)lock using Equation 
1. For each block, the FN sequence is then cyclically 
shifted in the opposite direction by one frequency coef- 
ficients, and the average over all the blocks is then conf>- 
puted. In practice, this process can be conputed 
incrementally and does not require temporary storage 
of all the extracted watermarks. A weighted averaging 
can also be applied, where the weights are determined 
based on their susceptibility to common signal transfor- 
mations such as low pass filtering. Rnally. the average 
watermark is compared with the original PN sequence 
via correlation. The reason for shifting the watermark in 
the column direction may now be apparent If the image 
is clipped on an artxtrary block tXHindary, then the com- 
puted average watermark will simply be rotated l>y an 
amount that is a function of the relative location of the 
clipped portion of the image. Correlation can then be 
performed on all permutatior^ (typically 1 1 to 25) of the 
watermark. The output from the correlator with the max- 
imum value is then used for decision purposes. The 
extraction stage is depicted in Rgure 4. Taking the max- 
imum correlator output over all rotations of the water- 
mark can cause the decision circuitry to be noisy. To 
improve this, the watermark is broken into two pieces; a 
synchronisation portion Is of length K and a verification 
portion Is N-K. Then, when the watermark is extracted 
as before, correlation Is first performed only on all rota- 
tions of the synchronization portion of thte watermark. 
The maximum con-elation output Is noted, then tile veri- 
fication portion of the watenrark is rotated by the corre- 
sponding amount and a second con-elation Is performed 
on the verification portions of the watermarks. This 
process signrficantiy improves the overall reliability of 
the system. In the course of experimentation, it was 
noticed that some watermarks performed better than 
others on the same imagery. This was caused by varia- 
tion in the con-elation statistics t>etween the synchroni- 
zation and verification portions of the watermark. 
Ideally, the two portions should have very low conela- 
tions. However, in several cases where watermarks per- 
formed poorly, it was traced to unexpected correlations 
between the two portions. 

The present invention provides a mocfification to 
digital watermarking methocte in which the original data 
is required for watermark extraction thereby enabling 
watermarking extraction in the absence of an unwater- 
marked or original data. The present invention prefera- 
bly uses MFEGAJPEQ coefficients. An image Is divided 



Into typically 8x8 block subimages or sii>regions and 
each subimage is processed and tfre results are com- 
bined to derive the extracted watermark. The result Is 
retraction of the watermark with very high confidence. 

5 While the above invention describes improvements 
to the prior-art INSERT-WHOLE, INSERT-MPEG-A. 
EXTRACT WHOLE, and EXTRACT-MPEG-A algo- 
rithms, It should be apparent to anyone skilled in the art 
that the same improvements may t>e applied to any 

10 algorithm for Inserting and extracting watermarks in 
image data. This more general view of the present 
invention is shown in Figures 8 and 9. 

Rgure 8 shows a schematic biock diagram of the 
general method for Inserting watermarte. This general 

IS method makes i^e of a non-bkx:k-based watermark 
Insertion algorithm, which shall be referred to hereafter 
as the l>ase insertion algorithm". The watermark 
erxxxler 80 converts the watermark into a fomn appro- 
priate for the base insertion algorithm. If the base inser- 

20 tion algorithm Is that shown in Rgure 1 . for example, 
then the watermark encoder 80 corresponds to the 
watermark encoder 19. which compr^es the enra cor- 
rection encoder 10. the PN mapper 11. and the spectral 
transformer 12. However. If a different base insertion 

25 algorithm is to be used, then the watermark encoder 80 
may perform a different transformation of the water- 
mark. The encoded watermark signal from watermark 
encoder 80 is provided as an input to watermark seg- 
menter 81, which divides the watermark into a set of 

30 subwatermaris. Any portion of tiie original watermark 
might appear redundantly in several of the resulting 
subwatermarks. The data to be watermari^ is pro- 
vided as an input to data segmenter 82, which divides 
the data into subregions. Each subwatermark is 

35 inserted into a respective data subregion by a water- 
mark Inserter 83a. 83b, etc. The watermark inserters 
inrplement the base insertion algorithm, so, if the base 
insertion algorithm is that shown in Rgure 1, then each 
watermark inserter 83a. 83b, etc.. corresponds to the 

40 watermark inserter 18, which comprises a spectral 
transformer 13. a spectral shaper 14. a delay 15, a sum- 
mer 16. arxj an inverse transform 17. Hcswever. if a dif- 
ferent base insertion algorithm Is to be used, then the 
watermark inserters 83a, 83b, etc., may enrploy a differ- 

45 ent method of inserting sutiwatermarks Into the subre- 
gions of the data to be watermarked. The outputs from 
the watermark inserters are assembled in data com- 
biner 84 to provide watermarked data. 

Rgure 9 shows a schematic t)lock diagram of the 

50 corresponding general exb-action algorithm. The algo- 
rithm makes use of a "t>ase extraction" algorithm that 
corresponds to the base insertion algorithm used in 
inserting the watermark Into the data to t>e watermarked 
(Rgure 8). The algorithm in Rgure 9 is sut>stantially the 

55 same as the algorithm shown in Rgure 4, except that in 
the general case, the spectrum normalizers 41a, etc. 
are replaced t}y watermark extractors 91a. etc., which 
implement the base extraction algorithm. That is. if the 
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base insertion algorithm used was the algorithm shown 
in Rgure 1. then the watermark extractors 91a. etc., in 
Rgure 9 will t>e the spectrum normalizers 41a. eta in 
Rgure 4. 

While there has been descrft>ed and illustrated a s 
system for inserting a watermark into and extracting a 
watermark from watermarked data without using an 
unwatermarked version of the data, it will be apparent to 
those skilled in the art tfiat variations and modifications 
are possible without deviating from the broad principles w 
and teachings of the present invention which shall be 
limited solely by the scope of the claims appended 
hereto. 

Clain^ 75 

1 . A method for inserting a watermark signal into data 
to be watermarked comprising the steps of: 

dividing data to be watermarked into a plurality 20 
of subregions; 

computing frequency coefficients of the data to 
be watermarked in each subregion; 
spread spectrum modulating a watermark sig- 
nal to t>e inserted by mapping the watermark 2S 
signal into a PN (pseudo-random noise) 
sequence; 

spectral shaping the PN sequence as a func- 
tion of the average power in each frequency 
coefficient of the data; and 30 
inserting each spectral shaped PN sequence 
into predetermined coefficients in the data in 
each subregion. 

2. A method for inserting a watermark signal into data 3S 
to be watermarked as set forth in daim 1, where 
said frequency coefficients are DCT (discrete 
cosine transform) coefficients. 

3. A method for inserting a watermark signal into data 4C 
to be watermari^ as set forth in claim 1 or 2, 
where each subregion is a 8 x 8 block of pixels. 

4. A method for inserting a watermark signal into data 

to be watermarked as set forth in claim S.where the 45 
frequency coefficients of the watermark signal are 
rotated prior to inserting of each spectral shaped 
PN sequence into the subregion. 

5. A method for inserting a watermark signal into data so 
to be watermarked as set forth in daim 4, where 
only a sut>set of the watermark signal frequency 
coefftdents is inserted into any one subregion. 

6. A method for inserting a watermark signal into data 55 
to be watermarked as set forth in daim 5 , where 
the watermark signal conprises a synchronization 
portion and a verification portion. 



7. A method for inserting a watermark signal into data 
to be watermarked as set forth in arry of daims 1 to 
6, where said inserting is performed after the data 
und^oes MPEG quantization processing. 

.8. A method for inserting a watermark signal into data 
to be watermarked as set forth in daims 6 or 7, 
where the synchronisation portion and the verifica- 
tion portion have very little correlation t>e!ween 
each other. 

9. A method for inserting a watermark signal into data 
to be watermarked as set forth in any of daims 1 to 
8. 

where the spectral sfiaping as a function of the 
average power is typically 3x1 window of the 
coefiident obtained from the one<limensional 
vectorization by zigzagging of two-dimension 
frequency ooeff idents. 

10. A method for inserting a watermark signal into data 
to be watermarked as set forth in any of daims 1 to 
8, where the spectral shaping is a function of the 
average power based on the two four-connected 
frequency coeffidents dosest to the DC term. 

11. A method of extracting a watermark from water- 
marked data cofTf}rising the steps of: 

receiving subregions of watermarked data; 
spectrum normalizing the watermarked data 
from each subregion to generate respective 
normalized signals; 

combining the respective normalized signals 
from each subregion to generate a single 
watermark; 

correlating the single watermark with predeter- 
mined PN (pseudo-random noise) sequences 
corresponding to predetermined symbols to 
provide correlated signals for each predeter- 
mined PN sequence in each subregion; 
deciding which correlated signal is rrxxst likely a 
current symbol; and extracting a sequence of 
most likely current symbds corresponding to 
the watermark. 

12. A method of extracting a watermark from water- 
marked data as set forth in claim 1 1 , where the sut)- 
regions are 8 x 8 blocks used for MPEG encoding 
and decoding. 

13. A method of extracting a watermark from water- 
marked data as set forth in daim 12, where said 
combining the normalized signals from each subre- 
gion to generate a single watermark, induding 
renxjving the relative rotation of the watermark 
between t>locks. 
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14. A method of extracting a watermark from water- 
marked data as set forth in claim 13, further com- 
prising subsequentty reconstructing the watermark 
from partial watermarks inserted into each tkxk. 

5 

15. A method of extracting a watermark from water- 
marked data as set forth In datm 14. further conv 
prising weighting the watermark coefficients tsased 
on their kx;ation within the frequency spectrum, 
where the weighting is a function of the susceptit>il- io 
ity of each frequency coefficient to common signal 
transformations. 

16. A method of extracting a watermark from water- 
marked data as set forth in daim 1 1 , further com- is 
prising correlating with all rotational shifts of the 
extracted watermark and selecting the maximum 
valua 

17. A method of extracting a watermark from water- so 
marked data as set forth in daim 11. further conv 
prising correlating with all rotational shifts of a 
synchronization portion of a watermark to deter- 
mine a maximum value and sutssequentiy rotating a 
verification portion of the watermark by the same 2S 
amount as the syndvonization portion ^ rotated to 
obtain the maximum value prior to correlating 
between the verification portion arxJ predetermined 

PN sequerK^es. 

30 

18. A method of extracting a watermark from water- 
marked data comprising the steps ot. 

receiving subregions of watermarked data; 
spectrum normalizing the watermarked data 35 
from each subregion to generate respective 
normalized signals; 

correlating tiie respective normalized signals 
with predetermined PN sequences corre- 
sponding to predetermined symbols to provkJe 4o 
con^elated signals for each predetermined PN 
sequence in each subregion; 
dedding which correlated signal is most likely a 
current symbol in each subregion for providing 
an extracted symbol stream; 45 
error correcting the extracted symbol stream; 
and 

extracting a sequence of most likely current 
symt>ds conesponding to the watermark. 

50 

19. A method of extracting a watermark from water- 
marked data as set forth in daim 18. where said 
error correction is Reed Solomon error correction. 

20. A method for ir^erting a watermark signal into data S5 
to be watermarked comprising tiie steps of: 

dividing data to be watermarked into a plurality 



of Sii>regions; 

dividing a watermark signal into a plurality of 
sut>watermarks where portions of the water- 
mark are contained in nmre than one sutTwater- 
mark; and 

inserting said plurality of subwatermarks into 
said plurality of subregior^. 

21 . A method for inserting a watermark signal Into data 
to t>e watermarked as set forth in daim 20, where 
each subwatermark is inserted into a respective 
subregion. so that each subregion contains at least 
one sid>watermark. 

22. A method for extracting a watermark signal from 
watermarked data comprising the steps of: 

receiving a plurality of subregions of watermark 
data; 

extracting a subwatermark from each subre- 
gion of said plurality of subregions; and 
corTt)ining arvJ averaging the subwatermarks 
extracted from all the sutxegions to obtain a 
signal conrvnensurate with the watermark sig- 
nal. 

23. A method for extracting a watermark signal from 
watermarked data as set forth in daim 22. furtiier 
comprising the steps of: 

dividing the signal commensurate with the 
watermark signal into a plurality of symbol sig- 
nals; 

correlating each symbol signal with a set of 
predefined signals; 

determining which predefined signal best cor- 
responds to each symbol signal; and 
combining the best corresponding predeter- 
mined signals to generate the watermark sig- 
nal. 



9 



EP0 840513A2 



WATERMARK 
SIGNAL 



WATERMARK 
ENCODER 



] 9 



1 0 



ERROR 
CORRE 
ENCOD 


CTION 
ER 


> 


. } 


PN MAPPER 


> 


. 5 


SPECTRAL 
TRANSFORMER 



1 1 



1 2 



FIQ 1 



DATA TO BE 
WATERMARKED 



! WATERMARK 
INSERTER 



1 8 



SPECTRAL 
TRANSFORMER 



3 



1 4 



I. 



1 5 



SPECTRAL 
SHAPER 



DELAY 



^1 6 



INVERSE 
TRANSFORM 



^1 7 



i ; 



WATERMARKED 
DATA 



10 



EPO 840513 A2 



WATERMARK 
SIGNAL 



2 0 



ERROR 

CORRECTION 
ENCODER 



2 1 



PN MAPPER 



2 2 



SPECTRAL 
TRANSFORMER 



i i 



2 3 



WATERMARK 
SEGMENTER 



DATA TO BE 
WATERMARKED 



i 



DATA 

SEGMENTER 



J 



WATERMARK 
INSERTER 



^2 5 a 



TTT 



2 5b 



WATERMARK 
INSERTER 



2 4 



2 5 i 



WATERMARK 
INSERTER 



FIG 2 



4 1 V 



DATA 

COMBINER 



^2 6 



T 



WATERMARKED 
DATA 



11 



EP0840513A2 



WATERMARKED 
DATA 



i 



SPECTRUM 
NORMAL I ZER 



3 1 a 



CORRELATOR 



3 1 b 



CORRELATOR 



DECISION 
CIRCUIT 



ERROR 
CORRECTOR 



^3 0 



WATERMARK 
DENTIFIERI 



3 1 



CORRELATOR 



3 2 



T 



^\^3 3 



FIG 3 



EXTRACTED 
WATERMARK 
SIGNAL 



12 



EP0840513A2 



WATERMARKED 
DATA 



1 



j: 



DATA 

SEGMEMTER 

TTT 



SPECTRUM 
NORMAL I ZER 



1 a 



0 



SPECTRUM 
NORMAL IZER 



4 1 b 



SPECTRUM 
NORMAL I ZER 



Li. i i i ^ 



WATERMARK 
COMBINER 



4 2 



SYMBOL 
SE PARATOR 

TTT 



WATERMARK 
IDENTIFIER 



^4 4 a 



^4 3 



WATERMARK 
IDENTIFIER 



4 b 



WATERMARK 
IDENTIFIER 



Q 



FIQ 4 



SYMBOL 
COMBtNER 



^4 5 



ERROR 
CORRECTOR 



4 6 



4 I i 



4 4 i 



13 



EP0840513A2 




FIG 6 



0 


1 


2 


3 




1 


2 


3 


4 




2 


3 


4 


5 




3 


4 


5 


6 





dc 



FIG 7 



14 



EP0840513A2 



WATERMARK 
SIGNAL 



DATA TO BE 
WATERMARKED 



8 0 



WATERMARK 
ENCODER 



i 



DATA 

SEGMENTER 



8 1 



WATERMARK 
SEGMENTER 



J 



WATERMARK 
INSERTER 



3 a 



TTT 



8 2 



8 3b 



WATERMARK 
INSERTER 



8 3 j 



WATERMARK 
INSERTER 



FIG 8 




WATERMARKED 
DATA 



15 



EP0 840513A2 



£ 



WATERMARK 
EXTRACTOR 



j: 



WATERMARK 
IDENTIFIER 



WATERMARKED 
DATA 



1 



DATA 

SE6MENTER 



J 



9 t a 



IT 







WATEF 
EXTR/ 


(MARK 
^CTOR 







^4 0 



^9 1 b 



^ '1' 'i' J" 



WATERMARK 
EXTRACTOR 



WATERMARK 
COMBINER 



^4 2 



J 



SYMBOL 
SE PARATOR 

rrr 



^4 4 a 



r 



^4 3 



WATERMARK 
IDENTIFIER 



4 4b 



WATERMARK 
IDENTIFIER 



Ir ^ r 4 Jr Jr ^ 



FIG 9 



SYMBOL 
COMBINER 

ERROR 
CORRECTOR 



^4 5 

/-v^4 6 



16 



This Page is Inserted by IFW Indexing and Scanning 
Operations and is not part of the Official Record 

BEST AVAILABLE IMAGES 

Defective images within this document are accurate representations of the original 
documents submitted by the applicant. 

Defects in the images include but are not limited to the items checked: 

□ BLACK BORDERS 

□ IMAGE CUT OFF AT TOP, BOTTOM OR SIDES 

□ FADED TEXT OR DRAWING 

□ BLURRED OR ILLEGIBLE TEXT OR DRAWING 

□ SKEWED/SLANTED IMAGES 

□ COLOR OR BLACK AND WHITE PHOTOGRAPHS 

□ GRAY SCALE DOCUMENTS 

□ LINES OR MARKS ON ORIGINAL DOCUMENT 
^Q^FERENCE(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY 

□ OTHER: 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 



